feat(review): pure config-drift evaluator over the loosenable-knob registry by RealDiligent · Pull Request #8232 · JSONbored/loopover

RealDiligent · 2026-07-23T13:25:02Z

Summary

Closes calibration: pure config-drift evaluator — does any alternative setting Pareto-dominate the live value on the trailing corpus? #8212
The loosening evaluator (evaluateKnobLoosening, src/services/loosening-knobs.ts) answers "can we safely loosen?"; nothing answered the inverse operator question from the calibration: backfill phase 2 — GitHub-truth reversed labels (retro successor scan) + raw-context re-fetch #8170 retro's largest identified wrongness source — stale configuration: "is what is CURRENTLY live still the best-supported setting, in either direction?"
Adds evaluateKnobDrift(knob, cases, liveValue) exported beside LOOSENABLE_KNOBS, mirroring evaluateKnobLoosening's discipline verbatim (the two named anchors): the knob's own split seed/fraction via splitBacktestCorpus, the same Pareto floor via compareBacktestScores (strictly improved on the visible split AND non-regressed on the deterministic held-out split), the same never-on-noise sample minimums, and the hard minimum no evidence may cross.
The candidate pool is every registry candidate plus the shipped value (a TIGHTER alternative dominating live is exactly the stale-config signal) minus the live value itself, tried nearest-to-live first — the minimal config change wins, mirroring smallest-step-first; equidistant ties deterministically prefer the tighter value. The report distinguishes direction: "shipped" (checked first — a drifted override should revert), "looser" (informational; duplicates the loosening loop's own proposal), "tighter" (actionable staleness warning), and carries corpus sizes plus both split comparisons per the Epic: validated auto-loosening for auto-tune.ts's tuning advisor, gated on a clean backtest #8121 evidence-trail convention.
Null — never a guess — on sample-floor misses or when nothing strictly dominates. Pure evaluation only: no cron, no alert, no writes, no new dependencies (per the issue's Boundaries).

Scope

The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

If any required check was skipped, explain why:

Ran the module's full suite (vitest run test/unit/loosening-knobs.test.ts — 14 tests green, including the 7 new drift cases mirroring the suite's membership-probe seeding technique per the issue's test requirement) plus the full root npm run typecheck and npm run build --workspace @loopover/engine. The new tests cover dominance in both directions, the shipped-value revert signal, the no-dominance null, the sample-floor null, held-out regression rejection, and byte-identical determinism — both sides of every conditional in the new evaluator. actionlint/workers/mcp/ui checks are untouched surfaces; CI runs them all.

Safety

No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
API/OpenAPI/MCP behavior is updated and tested where needed.
UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Not applicable — pure backend evaluation function (no UI, docs, or extension surface touched; no cron, no writes, nothing visible changes until the separate wire issue consumes it).

Notes

evaluateKnobLoosening itself is untouched — the drift evaluator sits beside it sharing the private compareOnSlice helper, so the two disciplines can never diverge on how a candidate is scored.

superagent-security · 2026-07-23T13:25:47Z

Superagent didn't find any vulnerabilities or security issues in this PR.

…gistry (JSONbored#8212) The loosening evaluator answers "can we safely loosen?"; nothing answered the inverse operator question from the JSONbored#8170 retro's largest wrongness source -- stale configuration: "is what is CURRENTLY live still the best-supported setting, in either direction?". Add evaluateKnobDrift(knob, cases, liveValue) beside evaluateKnobLoosening, mirroring its discipline verbatim: the knob's own split seed/fraction via splitBacktestCorpus, the same Pareto floor via compareBacktestScores (strictly improved on visible AND non-regressed on held-out), the same never-on-noise sample minimums, and the hard minimum no evidence may cross. The candidate pool is every registry candidate PLUS the shipped value -- a TIGHTER alternative dominating live is exactly the stale-config signal -- minus the live value itself, tried nearest-to-live first (minimal config change wins, mirroring smallest-step-first; equidistant ties deterministically prefer the tighter value). The report distinguishes direction ("shipped" checked first: a drifted override should revert; else looser = informational duplicate of the loosening loop, tighter = actionable staleness warning) and carries corpus sizes plus both split comparisons per the JSONbored#8121 evidence-trail convention. Null -- never a guess -- on sample-floor misses or when nothing strictly dominates. Pure evaluation only: no cron, no alert, no writes. Tests mirror the suite's membership-probe seeding: dominance in both directions, the shipped-value revert signal, no-dominance null, sample-floor null, held-out regression rejection, and byte-identical determinism.

…ub-hard-minimum filter (JSONbored#8212)

codecov · 2026-07-23T13:37:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.55%. Comparing base (0c8d3d6) to head (e52fbf6).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8232      +/-   ##
==========================================
- Coverage   92.10%   89.55%   -2.55%     
==========================================
  Files         776       98     -678     
  Lines       78328    22651   -55677     
  Branches    23668     3876   -19792     
==========================================
- Hits        72147    20286   -51861     
+ Misses       5062     2187    -2875     
+ Partials     1119      178     -941

Flag	Coverage Δ
shard-1	`93.10% <100.00%> (+38.17%)`	⬆️
shard-2	`51.72% <0.00%> (-0.88%)`	⬇️
shard-3	`3.44% <0.00%> (-51.32%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/services/loosening-knobs.ts	`100.00% <100.00%> (ø)`

... and 678 files with indirect coverage changes

loopover-orb · 2026-07-23T13:38:57Z

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-23 13:39:38 UTC}

2 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

safe to merge

Review summary
Adds evaluateKnobDrift, a pure sibling to evaluateKnobLoosening that checks whether the live value of a knob is still Pareto-optimal in either direction (not just whether it can be safely loosened). The candidate pool (shipped value + registry candidates, minus live, nearest-first with tighter-wins on ties) and split/floor discipline correctly mirror the existing loosening evaluator, and I traced the tie-break and 'shipped' direction logic against the test fixtures — both check out (e.g. tie_probe's 0.9 vs 0.85 equidistant case resolves to 0.9 as coded). This is pure evaluation code with no writes, no cron, no new deps, consistent with the issue's stated boundaries and closing #8212.

Nits — 3 non-blocking

src/services/loosening-knobs.ts:160-181 — the JSDoc and inline comment on KnobDriftReport/evaluateKnobDrift are fairly verbose relative to the sibling evaluateKnobLoosening's docs; consider trimming to match house style once this pattern is used elsewhere.
External brief flagged '8212' as an unexplained magic number at loosening-knobs.ts:160 — it's an issue-reference in a comment, not a literal used in logic, so no action needed.
Consider whether evaluateKnobDrift's shared logic with evaluateKnobLoosening (split/floor check, compareOnSlice loop) could be factored into a small shared helper now that there are two near-identical scan loops, though this is optional given how deliberately the PR mirrors the original for auditability.

Decision drivers

✅ Code review — No blockers (1 reviewer)
✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8212
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 343 registered-repo PR(s), 136 merged, 36 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	RealDiligent; Gittensor profile; 343 PR(s), 36 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: significant

Linked issue satisfaction

Addressed
The PR adds a pure evaluateKnobDrift function beside evaluateKnobLoosening that evaluates every registry candidate plus the shipped value, applies the same split/Pareto floor/sample-minimum discipline, distinguishes looser/tighter/shipped direction, and returns null on sample-floor misses or non-dominance, with corresponding tests covering both directions, no-dominance, sample-floor nulls, held-ou

Review context

Author: RealDiligent
Role context: outside_contributor
Public audience mode: oss maintainer
Lane context: Repository is configured for direct PR review.
Public profile languages: Python, Ruby, JavaScript, Svelte, TypeScript, Markdown, MDX, Rust
Official Gittensor activity: 343 PR(s), 36 issue(s).
PR-specific overlap: none found.

Contributor next steps

Keep the PR focused and include validation evidence before maintainer review.

Signal definitions

Related work = same linked issue, overlapping active PRs, or title/path similarity.
Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

@loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
@loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

Re-run LoopOver review

loopover-orb

LoopOver approves — the gate is satisfied and CI is green.

RealDiligent requested a review from JSONbored as a code owner July 23, 2026 13:25

RealDiligent added 2 commits July 23, 2026 21:28

test(review): cover the drift evaluator's equidistant tie-break and s…

e52fbf6

…ub-hard-minimum filter (JSONbored#8212)

loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 23, 2026

loopover-orb Bot assigned RealDiligent Jul 23, 2026

loopover-orb Bot approved these changes Jul 23, 2026

View reviewed changes

loopover-orb Bot merged commit 5bf6efe into JSONbored:main Jul 23, 2026
12 checks passed

cleanjunc mentioned this pull request Jul 23, 2026

feat(calibration): config-drift section in the maintainer recap (#8214) #8257

Closed

24 tasks

github-actions Bot mentioned this pull request Jul 23, 2026

chore(release): cut orb-v3.3.0 #8268

Merged

Navigation Menu

Uh oh!

feat(review): pure config-drift evaluator over the loosenable-knob registry - #8232

feat(review): pure config-drift evaluator over the loosenable-knob registry#8232
loopover-orb[bot] merged 2 commits into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-knob-drift-8212

RealDiligent commented Jul 23, 2026

Uh oh!

superagent-security Bot commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026

Uh oh!

loopover-orb Bot commented Jul 23, 2026 •

edited

Loading

✅ LoopOver review result - approve/merge recommended

Uh oh!

loopover-orb Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

RealDiligent commented Jul 23, 2026

Summary

Scope

Validation

Safety

UI Evidence

Notes

Uh oh!

superagent-security Bot commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026

Codecov Report

Uh oh!

loopover-orb Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ LoopOver review result - approve/merge recommended

Uh oh!

loopover-orb Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

loopover-orb Bot commented Jul 23, 2026 •

edited

Loading